Smarsh Pioneers Automated Compliance and Security Audits for Agentic AI Workloads Using Custom Claude Skills on AWS

The rapid integration of generative artificial intelligence into heavily regulated industries has created an unprecedented governance challenge. While consumer-facing applications often adopt a "move fast and break things" methodology, financial institutions, healthcare providers, and multinational corporations operating in highly supervised sectors cannot afford compliance failures. When sensitive customer communications data is processed by autonomous systems, security and auditability are non-negotiable. Addressing this critical requirement, Smarsh, a global leader in electronic communications archiving and digital governance, has successfully conducted an evidence-based review of its pioneering multi-agent workload. Utilizing the AWS Well-Architected Framework’s newly introduced Agentic AI Lens alongside a custom-built skill for Anthropic’s Claude assistant, Smarsh has redefined how complex autonomous artificial intelligence systems are evaluated against rigorous industry standards.

The Background and Governance Landscape of Digital Communications

For more than two decades, Smarsh has served as a cornerstone of digital compliance for over 7,000 organizations, ranging from emerging enterprises to multinational financial powerhouses. The company’s core mission is to enable these entities to capture, retain, monitor, and analyze electronic communications across more than 100 distinct channels—including email, mobile messaging, voice calls, video conferences, enterprise chat platforms, and emerging AI assistants. Operating at this scale requires absolute trust in the underlying infrastructure, which relies heavily on Amazon Web Services (AWS). Recognizing its operational excellence, AWS named Smarsh a 2025 AWS Partner of the Year Award winner, cementing a deep, long-standing technological collaboration.

As enterprise technology shifts from traditional software applications to autonomous, multi-agent artificial intelligence networks, the regulatory perimeter expands exponentially. Smarsh’s next-generation artificial intelligence capabilities are constructed on Amazon Bedrock AgentCore, an advanced platform designed to build, connect, and optimize autonomous agents at scale regardless of the underlying framework or model. By leveraging AgentCore’s robust architecture—spanning its runtime, gateway, registry, and identity management capabilities—Smarsh can deploy sophisticated compliance monitoring agents. However, deploying these workloads in production requires navigating a labyrinth of regulatory mandates, security protocols, and internal governance frameworks to ensure every autonomous decision can be audited, validated, and proven compliant before reaching an end customer.

The Operational Challenge: Navigating Un-tooled Prescriptive Guidance

The primary hurdle faced by the Smarsh architectural team stemmed from a common friction point in enterprise cloud adoption: the velocity at which prescriptive cloud guidance evolves often outpaces the availability of native console tooling. When a Smarsh architect set out to review the company’s first multi-agent workload—an intelligent agent review system designed to process complex compliance alerts—the natural choice was the AWS Well-Architected Framework’s Agentic AI Lens.

Unlike more mature offerings, such as the Serverless Applications Lens, which can be selected and executed directly within the AWS Well-Architected Tool console, the Agentic AI Lens debuted strictly as published documentation. Consisting of 150 granular best practices distributed across 41 questions and six foundational pillars, the lens outlines specific risk levels and desired outcomes for autonomous systems. However, because it lacked immediate native console support, organizations faced a difficult dilemma. Relying on manual evaluations would necessitate a multi-day review process where different architectural reviewers structured assessments inconsistently. Alternatively, waiting for future console integrations would introduce unacceptable delays to product deployment schedules. Refusing to compromise between rigorous compliance and agile delivery, Smarsh engineered an innovative third alternative: making the Agentic AI Lens executable through automation.

Solution Architecture: Transforming Static Lens Documentation into an Executable Skill

To operationalize the review process efficiently, Smarsh packaged the entire evaluation methodology into a custom-built skill for Claude, Anthropic’s advanced AI assistant. Designed as a versioned bundle containing structured instructions, reference datasets, and executable scripts, the skill leverages frontmatter descriptions to remain dormant until explicitly invoked. This design ensures optimal efficiency through progressive disclosure, maintaining near-zero token costs until a comprehensive workload review is formally requested.

The architecture of the Smarsh assessment skill rests upon three core components that bridge raw repository data with the structured taxonomy of the AWS Well-Architected Framework. The operational workflow is governed by a strict assessment contract maintained in a standardized JSON data format. This ensures that every evaluation yields objective, uniform outputs. For example, specific lens items—such as input validation protocols under security identifier AGENTSEC08-BP01—are programmatically audited, categorized by status (such as "Partially Compliant"), supported with direct evidence from the codebase, and paired with actionable remediation steps.

Executing an audit is streamlined into a single interactive request. An engineer or architect simply points Claude at a designated code repository and requests a Well-Architected review. The skill then executes a methodical seven-step workflow: parsing the repository’s infrastructure definitions, analyzing continuous integration pipelines, reviewing authentication and gateway configurations, evaluating test suites, mapping findings against the 150 best practices of the Agentic AI Lens, generating a detailed remediation backlog, and compiling a comprehensive review workbook.

Rigorous Validation and Empirical Evaluation of the Assessment Skill

Before placing institutional trust in the automated review tool, the Smarsh engineering team subjected the custom Claude skill to rigorous internal validation and regression testing. The objective was to prove that the automation could consistently produce accurate, objective, and comprehensive audits without hallucinating compliance statuses or overlooking critical vulnerabilities.

To test the system, the team constructed two synthetic fixture repositories. The first was a moderately mature LangGraph-style workload representing standard industry practices, while the second was a deliberately flawed proof-of-concept containing hardcoded credentials and a historical record of runaway computational costs. The team executed multiple evaluation runs both with and without the custom skill enabled, grading each output against eight verifiable criteria per fixture. These criteria required the presence of all 150 lens items, the absence of blank statuses, mandatory evidence citations for every definitive finding, honest calibration, and the successful identification of fixture-specific anomalies like the embedded credentials.

The empirical results were definitive. Runs utilizing the custom skill achieved a flawless 16 out of 16 success rate across all evaluation metrics. In contrast, baseline runs executed without the skill passed only 8 out of 16 expectations, generating fragmented, ad-hoc checklists containing merely 18 to 39 items with zero traceability back to the AWS Well-Architected Lens. Because this evaluation suite is permanently embedded within the skill package, continuous regression testing occurs natively as the tool evolves.

Findings from the First Real-World Workload Review

Deploying the validated skill against its production-grade multi-agent compliance workload, Smarsh yielded profound insights into its system architecture. The skill systematically ingested the repository’s documentation, AWS Cloud Development Kit (AWS CDK) infrastructure definitions, GitHub Actions automation workflows, registry and gateway design records, and extensive test suites, completing a full evaluation of all 150 lens items in a matter of hours.

The resulting audit workbook delivered an objective balance of organizational strengths and critical security gaps. Among its primary strengths, the system verified per-agent AWS Identity and Access Management (IAM) execution roles strictly scoped to named runtime Amazon Resource Names (ARNs). Furthermore, it confirmed the use of SigV4-signed Model Context Protocol (MCP) calls routed securely through the AgentCore Gateway, schema-validated inputs equipped with typed error handling contracts, exportable decision audit trails, and continuous behavioral monitoring facilitated by Amazon Bedrock AgentCore online evaluations.

Conversely, the evaluation identified significant high-risk gaps requiring immediate mitigation. These included an ingress authorizer operating with authentication temporarily disabled under a formally documented risk acceptance, an absence of prompt-injection screening mechanisms designed to neutralize malicious payloads embedded in communications content, rate-limiting logic categorized as an unfinished engineering task, and a lack of automated cost-control safeguards. Each identified vulnerability was explicitly mapped to its corresponding lens item, supported by hard code evidence, and accompanied by a precise engineering remediation path.

Following the automated assessment, Smarsh architects engaged directly with engineering leadership to triage the findings. Teams validated statuses against institutional knowledge, adjusted priorities within the remediation backlog, and refined the architectural roadmap. Notably, several discrete vulnerabilities were consolidated into a unified anomaly-detection epic, while other findings were upgraded in priority after engineering pointed out overlooked testing mechanisms already active in pull request pipelines.

Elevating AWS Account Engagement Through Targeted Inquiries

One of the most consequential outcomes of the automated Well-Architected review was its ability to elevate the quality of technical discourse between Smarsh and its AWS account team. Certain complex architectural questions transcend what can be answered purely by analyzing a code repository—particularly those involving the foundational semantics and native behaviors of underlying cloud services.

Rather than scheduling a generalized, open-ended architecture review, Smarsh utilized the structured workbook to construct a highly focused, data-driven engagement agenda for its AWS Solutions Architects. Every discussion point was directly mapped to a specific checklist item required to close a compliance gap. This targeted approach transformed routine advisory meetings into highly productive working sessions characterized by strict definitions of done, accelerating the resolution of complex infrastructure challenges.

Broader Industry Implications and Future Strategic Roadmap

The methodology pioneered by Smarsh addresses a systemic industry challenge. As artificial intelligence governance frameworks advance faster than cloud console integrations can accommodate, enterprises require repeatable, automated mechanisms to audit autonomous systems. By treating compliance lenses as structured data, encoding evaluation logic into versioned assistant skills, validating tooling through rigorous benchmarking, and maintaining human oversight for strategic judgment, organizations can bridge the gap between rapid software innovation and strict regulatory compliance.

Looking ahead, Smarsh plans to expand the capabilities of its review framework. The company’s strategic roadmap includes embedding the assessment skill directly into continuous integration and continuous deployment (CI/CD) pipelines to evaluate agentic workloads automatically before pull requests are merged. Additionally, Smarsh aims to generalize the framework to support other emerging specialized lenses within the AWS Well-Architected ecosystem, further automating its multi-cloud and hybrid governance postures.

For organizations operating in regulated sectors, the lesson is clear. Navigating the era of agentic artificial intelligence demands proactive, systematic evaluation tools. By turning compliance frameworks into executable code, Smarsh has established a scalable benchmark for the secure deployment of autonomous systems, proving that speed and rigorous regulatory compliance are not mutually exclusive, but mutually reinforcing pillars of modern enterprise architecture.